{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Fraud Detector - Data Profiler Notebook \n",
    "\n",
    "\n",
    "### Dataset Guidance\n",
    "-------\n",
    "\n",
    "AWS Fraud Detector's Online Fraud Insights(OFI) model supports a flexible schema, enabling you to train an OFI model to your specific data and business need. This notebook was developed to help you profile your data and identify potenital issues before you train an OFI model. The following summarizes the minimimum CSV File requirements:\n",
    "\n",
    "* The files are in CSV UTF-8 (comma delimited) format (*.csv).\n",
    "* The file should contain at least 10k rows and the following __four__ required fields:  \n",
    "\n",
    "    * Event timestamp \n",
    "    * IP address \n",
    "    * Email address\n",
    "    * Fraud label \n",
    "    \n",
    "* The maximum file size is 10 gigabytes (GB).  \n",
    "\n",
    "* The following dates and datetime formats are supported:\n",
    "    * Dates: YYYY-MM-DD (eg. 2019-03-21)\n",
    "    * Datetime: YYYY-MM-DD HH:mm:ss (eg. 2019-03-21 12:01:32) \n",
    "    * ISO 8601 Datetime: YYYY-MM-DDTHH:mm:ss+/-HH:mm (eg. 2019-03-21T20:58:41+07:00)\n",
    "\n",
    "* The decimal precision is up to four decimal places.\n",
    "* Numeric data should not contain commas and currency symbols. \n",
    "* Columns with values that could contain commas, such as address or custom text should be enclosed in double quotes. \n",
    "\n",
    "\n",
    "\n",
    "### Getting Started with Data \n",
    "-------\n",
    "The following general guidance is provided to get the most out of your AWS Fraud Detector Online Fraud Insights Model. \n",
    "\n",
    "* Gathering Data -  The OFI model requires a minimum of 10k records. We recommend that a minimum of 6 weeks of historic data is collected, though 3 - 6 months of data is preferable.   As part of the process the OFI model partitions your data based on the Event Timestamp such that performance metrics are calculated on the out of sample (latest) data, thus the format of the event timestamp is important. \n",
    "\n",
    "  \n",
    "* Data & Label Maturity: As part of the data gathering process we want to insure that records have had sufficient time to “mature”, i.e. that enough time has passed to insure “non-fraud\" and  “fraud” records have been correctly identified. It often takes 30 - 45 days (or more) to correctly identify fraudulent events, because of this it is important to insure that the latest records are at least 30 days old or older.  \n",
    "\n",
    "  \n",
    "* Sampling: The OFI training process will sample and partition historic based on event timestamp. There is no need to manually sample the data and doing so may negatively influence your model’s results.  \n",
    "\n",
    "  \n",
    "* Fraud Labels:  The OFI model requires that a minimum of 500 observations are identified and labeled as “fraud”. As noted above, fraud label maturity is important. Insure that extracted data has sufficiently matured to insure that fraudulent events have been reliably found. \n",
    "  \n",
    "  \n",
    "* Custom Fields: the OFI model requires 4 fields: event timestamp, IP address, email address and fraud label. The more custom fields you provide the better the OFI model can differentiate between fraud and not fraud.  \n",
    "  \n",
    "  \n",
    "* Nulls and Missing Values: OFI model handles null and missing values, however the percentage of nulls in key fields should be limited. Especially timestamp and fraud label columns should not contain any missing values.   \n",
    "\n",
    "  \n",
    "If you would like to know more, please check out the [Fraud Detector's Documentation](https://docs.aws.amazon.com/frauddetector/). \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>.container { width:90% }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "from IPython.display import clear_output\n",
    "display(HTML(\"<style>.container { width:90% }</style>\"))\n",
    "from IPython.display import IFrame\n",
    "# ------------------------------------------------------------------\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_rows', 500)\n",
    "pd.set_option('display.max_columns', 500)\n",
    "pd.set_option('display.width', 1000)\n",
    "pd.options.display.float_format = '{:.4f}'.format\n",
    "\n",
    "# -- AWS stuff -- \n",
    "import boto3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Amazon Fraud Detector Profiling \n",
    "-----\n",
    "\n",
    "from github download and copy the afd_profile.py python program and template directory to your notebook  \n",
    "\n",
    "<div class=\"alert alert-info\"> <strong> afd_profile.py </strong>\n",
    "\n",
    "- afd_profile.py - is the python package which will generate your profile report. \n",
    "- /templates - directory contains the supporting profile templates \n",
    "\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -- get this package from github -- \n",
    "import afd_profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Intialize your S3 client \n",
    "-----\n",
    "https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = boto3.client('s3')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File & Field Mapping\n",
    "-----\n",
    "Simply map your file and field names to the required config values. \n",
    "\n",
    "<div class=\"alert alert-info\"> <strong> Map the Required fields </strong>\n",
    "\n",
    "- input_file: this is your CSV file in your s3 bucket \n",
    "\n",
    "<b> required_features </b> are the minimally required freatures to run Amazon Fraud Detector \n",
    "- EVENT_TIMESTAMP: map this to your file's Date or Datetime field.    \n",
    "- IP_ADDRESS: map this to your file's IP address field.   \n",
    "- EMAIL_ADDRESS: map this to your file's email address field.  \n",
    "- FRAUD_LABEL: map this to your file's fraud label field.  \n",
    "    **note: the profiler will identify the \"rare\" case and assume that it is fraud**\n",
    "    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -- update your configuration -- \n",
    "config = {  \n",
    "    \"input_file\"        : \"<training dataset name>.csv\",\n",
    "    \"required_features\" : {\n",
    "        \"EVENT_TIMESTAMP\" : \"EVENT_DATE\",\n",
    "        \"EVENT_LABEL\"     : \"EVENT_LABEL\",\n",
    "        \"IP_ADDRESS\"      : \"ip_address\",\n",
    "        \"EMAIL_ADDRESS\"   : \"user_email\"\n",
    "    }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run Profiler\n",
    "-----\n",
    "The profiler will read your file and produce an HTML file as a result which will be displayed inline within this notebook.  \n",
    "  \n",
    "Note: you can also open **report.html** in a separate browser tab. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "# -- generate the report object --\n",
    "report = afd_profile.profile_report(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1500\"\n",
       "            height=\"800\"\n",
       "            src=\"schwab_training.html\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x127d7a518>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(\"report.html\", \"w\") as file:\n",
    "    file.write(report)\n",
    "\n",
    "IFrame(src='report.html', width=1500, height=800)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
